Superintelligence: Paths, Dangers, Strategies
- Downloads：2037
- Type：Epub+TxT+PDF+Mobi
- Create Date：2021-05-13 07:51:57
- Update Date：2025-09-06
- Status：finish
- Author：Nick Bostrom
- ISBN：0198739834
- Environment：PC/Android/iPhone/iPad/Kindle
- Download
Summary
A New York Times bestseller
Superintelligence asks the questions: What happens when machines surpass humans in general intelligence？ Will artificial agents save or destroy us？ Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life。
The human brain has some capabilities that the brains of other animals lack。 It is to these distinctive capabilities that our species owes its dominant position。 If machine brains surpassed human brains in general intelligence， then this new superintelligence could become extremely powerful - possibly beyond our control。 As the fate of the gorillas now depends more on humans than on the species itself， so would the fate of humankind depend on the actions of the machine superintelligence。
But we have one advantage: we get to make the first move。 Will it be possible to construct a seed Artificial Intelligence， to engineer initial conditions so as to make an intelligence explosion survivable？ How could one achieve a controlled detonation？
This profoundly ambitious and original book breaks down a vast track of difficult intellectual terrain。 After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life， we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time。
Superintelligence asks the questions: What happens when machines surpass humans in general intelligence？ Will artificial agents save or destroy us？ Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life。
The human brain has some capabilities that the brains of other animals lack。 It is to these distinctive capabilities that our species owes its dominant position。 If machine brains surpassed human brains in general intelligence， then this new superintelligence could become extremely powerful - possibly beyond our control。 As the fate of the gorillas now depends more on humans than on the species itself， so would the fate of humankind depend on the actions of the machine superintelligence。
But we have one advantage: we get to make the first move。 Will it be possible to construct a seed Artificial Intelligence， to engineer initial conditions so as to make an intelligence explosion survivable？ How could one achieve a controlled detonation？
This profoundly ambitious and original book breaks down a vast track of difficult intellectual terrain。 After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life， we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time。
Reviews
John Doez
，
Para ser un libro de divulgación científica， es poco esclarecedor。 No solo plantea ideas intrincadas que no consigue presentar de forma sencilla al lector， sino que da vueltas y vueltas alrededor de las mismas reflexiones cuando el tema de la IA （Inteligencia Artificial） parece ser mucho más amplio。 Además， uno esperaría un poco más de pensamiento filosófico al respecto。El libro es una advertencia para especialistas， pues dudo que llegue a un público generalista。 Para aquellos no tan versados en Para ser un libro de divulgación científica， es poco esclarecedor。 No solo plantea ideas intrincadas que no consigue presentar de forma sencilla al lector， sino que da vueltas y vueltas alrededor de las mismas reflexiones cuando el tema de la IA （Inteligencia Artificial） parece ser mucho más amplio。 Además， uno esperaría un poco más de pensamiento filosófico al respecto。El libro es una advertencia para especialistas， pues dudo que llegue a un público generalista。 Para aquellos no tan versados en IA， recomendaría ， mucho más accesible， con ideas más interesantes y un planteamiento más amplio。 。。。more
Pcbernhard
，
Often speculative and sligthly outdated， but still a good read abouth a possibly very dangerous development。 I do not fear the computers as such but one should fear the people behind them。
Michael Perkins
，
Before this book， I read the excellent book on A。I。， Human Compatible （link below）。Though Superintelligence came out first， I treated it as companion volume to Compatible。 This author explores several scary what-if A。I。 takeover scenarios that are not included in Compatible。 Here's the best review of this book by a true expert。。。。https://www。goodreads。com/review/show。。。My review of Human Compatible。。。https://www。goodreads。com/review/show。。。 Before this book， I read the excellent book on A。I。， Human Compatible （link below）。Though Superintelligence came out first， I treated it as companion volume to Compatible。 This author explores several scary what-if A。I。 takeover scenarios that are not included in Compatible。 Here's the best review of this book by a true expert。。。。https://www。goodreads。com/review/show。。。My review of Human Compatible。。。https://www。goodreads。com/review/show。。。 。。。more
Daniel Hrenak
，
Ak si myslíte， že ľudsto môže skončiť iba vďaka prírodnej katastrofe， globálnemu otepľovaniu alebo jadrovej vojne， tak môžete kľudne spávať。 Tých možností， ktoré reálne neovplyvníte je oveľa viac （takže načo sa tým vôbec trápiť） a jednou s nich je umelá inteligencia - AI na úrovni superinteligencie - teda niečo oveľa oveľa oveľa viac inteligentné ako my。 K vysvetleniu toho， čo to je， ako sa k tomu dopracovať， ako s tým zachádzať， ako tomu zadávať ciele， čo od toho vôbec očakávať a kvantá ďalších Ak si myslíte， že ľudsto môže skončiť iba vďaka prírodnej katastrofe， globálnemu otepľovaniu alebo jadrovej vojne， tak môžete kľudne spávať。 Tých možností， ktoré reálne neovplyvníte je oveľa viac （takže načo sa tým vôbec trápiť） a jednou s nich je umelá inteligencia - AI na úrovni superinteligencie - teda niečo oveľa oveľa oveľa viac inteligentné ako my。 K vysvetleniu toho， čo to je， ako sa k tomu dopracovať， ako s tým zachádzať， ako tomu zadávať ciele， čo od toho vôbec očakávať a kvantá ďalších otázok， ktoré vám （bez urážky） vôbec neprídu ani na myseľ， sa dopátrate práve tu。 Nie je to ľahké čítanie （či už témou alebo kvôli odborným termínom）， ale ak máte radi sci-fi， populárnu vedu a premýšľate čo čaká nás a naše deti v ďalších desaťročiach， mali by ste jej dať určite šancu。 Nedostanete žiadnu konkrétnu odpoveď ani predpoveď， iba smer， ktorým trebam upriamiť premýšľanie。 A to v tomto momente stačí。 Teda musí stačiť， pretože odpovede （či už sa nám budú páčiť alebo nie） prídu samé o pár mesiacov， rokov alebo （snáď až） desaťročí。 。。。more
Samih Ibrahim
，
Been a while since I've read a scientific book， and this is as scientific as they can get。 Filled with what-ifs， various scenarios analysis， and loads of jargon。 Nonetheless， an entertaining read in itself and should be a mandatory read for anyone working in AI research。 Been a while since I've read a scientific book， and this is as scientific as they can get。 Filled with what-ifs， various scenarios analysis， and loads of jargon。 Nonetheless， an entertaining read in itself and should be a mandatory read for anyone working in AI research。 。。。more
Rob Damon
，
I wanted some serious food for thought and an up-to-date account on the state of AI and I suppose I got what I wanted and yet I feel unsatisfied。 The book seems to lack a down to earth attitude and instead heads off on a "what if" tragectory， the same you can find in any Sci Fi story about AI。 There is an assumption made that in the future someone somewhere IS going to create an AI system that will create its own motivations and exhibit behavoiur that we cannot anticipate and will somehow bring I wanted some serious food for thought and an up-to-date account on the state of AI and I suppose I got what I wanted and yet I feel unsatisfied。 The book seems to lack a down to earth attitude and instead heads off on a "what if" tragectory， the same you can find in any Sci Fi story about AI。 There is an assumption made that in the future someone somewhere IS going to create an AI system that will create its own motivations and exhibit behavoiur that we cannot anticipate and will somehow bring about our downfall。 The Terminator， I Robot， 2001。。。。they all said it。 I think Arthur C Clarke should be remembered with his quote about humans reserving the right to pull the plug but Sci Fi writers， and the writer of this book， seem to assume humans will become so stupid that we'll delegate all our responsibilities to AI and NOT reserve the right to pull the plug。 Another issue with the book for me is what do we mean by Artificial Intelligence？ Do we mean a system that can beat everyone at Chess？ What if that same system was asked to play Poker， or Snap， would it be any good？ A first chapter exploring possible definitions of what we mean by "Intelligence" would have grounded this book a little。 。。。more
Vasyl X
，
Читав цю книгу не рівномірно。 Були захопливі моменти особливо на початку， коли автор розказує про можливі напрямки розвитку ШІ。 А були моменти коли читав через силу， особливо в розлогих філософських пасажах。 Можливо мій рівень володіння інформацією недостатній， але деякі проблеми， з якими пропонує боротися чи контролювати автор виглядають надуманими， або навпаки， з цілого спектру початкових проблем автор обирає тільки одну і заглиблюється в неї на кожному кроці， знову ж таки， обираючи тільки оди Читав цю книгу не рівномірно。 Були захопливі моменти особливо на початку， коли автор розказує про можливі напрямки розвитку ШІ。 А були моменти коли читав через силу， особливо в розлогих філософських пасажах。 Можливо мій рівень володіння інформацією недостатній， але деякі проблеми， з якими пропонує боротися чи контролювати автор виглядають надуманими， або навпаки， з цілого спектру початкових проблем автор обирає тільки одну і заглиблюється в неї на кожному кроці， знову ж таки， обираючи тільки один із можливих варіантів розвитку подій。 Книга дає хороший екскурс в історію розвитку ШІ。 Також дає розуміння можливих методів використання і досягнення ШІ。 Опис проблем які можуть виникнути у суспільства що досягнуло створення ШІ і трохи охолоджує інтузіазм з яким питомо людині хочеться приблизити такі винаходи。 Також дуже цікавою і потенційно правдивою є гіпотеза про те що як тільки ШІ досягне певного розвитку і людство не зуміє впровадити якісь методи впливу і контролю завчасно， то майбутнє людства， як такого， буде під загрозою。 。。。more
Colophōn
，
Garbage
Lorenzo LD
，
Too complex。 Nick Bostrom is great， but I think the purpose wasn’t educate people on AI。 The math formulas and the difficulty of the vocabulary mixed with the tech slang made the book really difficult to read。If you’re not a Softwares Engineer nor a Computer Programmer this book is not for you。Everything in this book is speculative。
Charlie Walker
，
I give this book a “Read and Pass it on” This might be one of the more difficult books I have read， but this book is for anyone that has any interest in Artificial Intelligence or the various scenarios that can bring around an existential crisis event to our species。 This is a very systematic book breaking down the many variables on our race to creating an AI capable of super-intelligence。 It does get confusing but so long as you understand the basic underlaying principles， this book quickly bec I give this book a “Read and Pass it on” This might be one of the more difficult books I have read， but this book is for anyone that has any interest in Artificial Intelligence or the various scenarios that can bring around an existential crisis event to our species。 This is a very systematic book breaking down the many variables on our race to creating an AI capable of super-intelligence。 It does get confusing but so long as you understand the basic underlaying principles， this book quickly becomes terrifying。 Having a global effort to safely build super-intelligent AI is right up there， in my estimation， of things we absolutely should worry about。 My ratings: ~ Better Than Sex！ ~ Must Have On The Shelf ~ Read And Pass It On ~ Struggle Through If You Must ~ Some Ol’ Bulls*^t 。。。more
Harshit Mathur
，
On the whole， this book has the right balance of technicality and exploration。 Bostrom has done an exemplary job of writing a book that can cater to a wide range of readers， from casual to the technical。The paperclip AI is just a crude， dumbed-down example of the very real possibility of an Artifical Intelligence program going crazy。Some reviewers have pointed out that most of Bostrom's scenarios belong to the realm of science fiction， not reality。 In doing this， they're questioning some seminal On the whole， this book has the right balance of technicality and exploration。 Bostrom has done an exemplary job of writing a book that can cater to a wide range of readers， from casual to the technical。The paperclip AI is just a crude， dumbed-down example of the very real possibility of an Artifical Intelligence program going crazy。Some reviewers have pointed out that most of Bostrom's scenarios belong to the realm of science fiction， not reality。 In doing this， they're questioning some seminal AI-related works put out by the brightest minds in the world， all of whom consider intelligence explosion a very real and imminent threat。I don't have any problem with the writing being too verbose at some places; on the contrary， the verbage helps convey Bostrom's points across clearly and convincingly。 。。。more
Eric Fritz
，
DNF 20%。 Somehow managed to be both too dense to read easily， and also provide no useful information。 It's complete conjecture and pessimistic for no reason。 DNF 20%。 Somehow managed to be both too dense to read easily， and also provide no useful information。 It's complete conjecture and pessimistic for no reason。 。。。more
Liam B。
，
A very insightful read， especially for anyone with interests in AI， philosophy， or the future of humanity in general。Nick Bostrom starts by convincing us of the possibility of superintelligence by proposing multiple different pathways it could arise （not all of which involve AI per se）。 From this Bostrom concludes even if there's only a small chance superintelligence could arise， it is still crucial to prepare for the consequences given the massive impact it could have。This book is dense。 It cov A very insightful read， especially for anyone with interests in AI， philosophy， or the future of humanity in general。Nick Bostrom starts by convincing us of the possibility of superintelligence by proposing multiple different pathways it could arise （not all of which involve AI per se）。 From this Bostrom concludes even if there's only a small chance superintelligence could arise， it is still crucial to prepare for the consequences given the massive impact it could have。This book is dense。 It covers a lot of ground very quickly - mainly in computer science and philosophy， but also other areas like economics and strategic planning。 I found Bostrom's writing to be clear and satisfying。 He presents ideas and arguments well。 That said， be prepared for a lot of new ideas and concepts to be introduced。 Bostrom is trying to paint a canvas of the problem and current progress （for 2014）， which necessitates a lot of information thrown at the reader。I have a number of takeaways from this book。 For one it further illuminated a topic I was interested in。 I hadn't considered the many different pathways to superintelligence， nor previous thought on how to actually build and influence a superintelligence。 This book takes a concept which was before mainly only considered in sci-fi and presents current analytical thought on it。 Another is the importance of not losing sight of the present。 It's easy to get swept away in future thinking with topics like this。 Indeed， I think it is important （and necessary） we work towards solutions to problems posed in this book。 However， we ought to be careful we don't get so wrapped up in them that we lose ourselves in the process。 。。。more
Michael Kraitsberg
，
Kind of modern theology。 At some point in the future superintelligent machine is likely to appear。 Then it may just kill us all， or worth doom to an infinite torment。 On the other hand， if we prepare wisely， it could solve all our problem， leaving humankind to do whatever the folks in the garden of Eden do。
Yazeed
，
It took me a while to think of a review because I frankly didn’t know how to feel about this book。 It revolves around AI which is a field I knew nothing about， I picked this book to gain a deeper understanding for that field。 I now realise that I picked the gospel of AI， the introduction here will give you a solid foundation for you to then branch into whichever subfield of AI。 I can’t recommend this book enough， if you’re interested in AI or have heard of it and want to know more， I can assure It took me a while to think of a review because I frankly didn’t know how to feel about this book。 It revolves around AI which is a field I knew nothing about， I picked this book to gain a deeper understanding for that field。 I now realise that I picked the gospel of AI， the introduction here will give you a solid foundation for you to then branch into whichever subfield of AI。 I can’t recommend this book enough， if you’re interested in AI or have heard of it and want to know more， I can assure you this book will cover most of the superficial knowledge of that field。The author gave the opportunity for readers to skip back and forth for optimal reading experience but I liked this book so much I read every word！ I will be referring back to it， because although AI is a fast developing filed this book is still and will always be relevant in my opinion。 。。。more
Stephen
，
Enjoyed the concepts discussed but would warn against listening to it on audiobook。
henry
，
The Meditations on First Philosophy of the field of artificial intelligence value alignment。
Andrey
，
This review has been hidden because it contains spoilers。 To view it， click here。 Finished that about a week ago。 Had to read it， because it is a modern classic on AI。A very difficult book to read （listen to）。 I think， for a "casual" reader， it is better to just read synapsis of it on wiki， or read an abridged version and be done with it。 Here is the essence of the book that I captured （in my own words）:1。 Sooner or later we will create superintelligence。 As soon as we reach the Artificial General Intelligence （AGI） threshold， by the whole human brain emulation or by other me Finished that about a week ago。 Had to read it， because it is a modern classic on AI。A very difficult book to read （listen to）。 I think， for a "casual" reader， it is better to just read synapsis of it on wiki， or read an abridged version and be done with it。 Here is the essence of the book that I captured （in my own words）:1。 Sooner or later we will create superintelligence。 As soon as we reach the Artificial General Intelligence （AGI） threshold， by the whole human brain emulation or by other means， it will quickly take off and will become superintelligent （SI）。2。 When we create it， even if it is SI in one narrow sense （for instance， an intelligent research machine）， it will quickly become SI in all other areas。3。 And then the default outcome is doom for all humanity 🙂 Because， whatever goals we set for SI， or SI sets for itself， it will not be limited by our set of values。4。 To avoid #3， we need to solve the value loading problem。 If we load values， before machine becomes SI， it will lack intelligence to understand them。 If we load values after， it will be too late， because we will lack intelligence to override AGI's values。It goes into great depth on each of those， and into a bunch of other topics。 Like， for instance: will it be a race of competing AGI projects or one AGI will rule them all; will it be better to invent AGI sooner or later？ When we invent AGI， how soon it will become SI？ Why the default outcome is doom and why SI will definitely be compelled to dispose of us and capture the whole universe？To summarize my view， I find it hard to believe #1 and #2， but this an important book because it tries to analyze #3 and #4。 So you're welcome and feel free to skip this book now 🙂 。。。more
Juan Manuel Cafferata
，
Mucha info， mucha sustancia， pero poco corazón。 Parece un libro escrito por una inteligencia artificial no tan sofisticada
Christopher Hudson Jr。
，
Compelling and unsettling。 Bostrom's writing style definitely takes some getting used to， but Superintelligence covers so much history， politics， technology， and philosophy in such a limited space， it makes up for the esoteric writing。 Would recommend to anyone interested in the potential risks associated with AI。 Compelling and unsettling。 Bostrom's writing style definitely takes some getting used to， but Superintelligence covers so much history， politics， technology， and philosophy in such a limited space， it makes up for the esoteric writing。 Would recommend to anyone interested in the potential risks associated with AI。 。。。more
Paul E。
，
This was recommended by a higher-up at the place where I worked a few years ago。 Interesting read about some realistic scenarios。（view spoiler）[Two memorable parts:1。 One section about the impracticality of downloading/uploading data directly to the brain。2。 A generally intelligent AI may not have the same goals as humans。 It may take over the world and then try to use all the Earth's resources to build as many paper clips as possible。（hide spoiler）] This was recommended by a higher-up at the place where I worked a few years ago。 Interesting read about some realistic scenarios。（view spoiler）[Two memorable parts:1。 One section about the impracticality of downloading/uploading data directly to the brain。2。 A generally intelligent AI may not have the same goals as humans。 It may take over the world and then try to use all the Earth's resources to build as many paper clips as possible。（hide spoiler）] 。。。more
Andreas
，
Artificial General Intelligence （AGI） will recursively improve itself， leading to a technological singularity and unpredictable changes to human civilization。 Low probability combined with high impact generates a risk which certainly makes one wonder about the backgrounds。Academic philosopher Nick Bostrom is by far not the first to argue about the singularity， but he makes a high effort to imagine how it would come so far， what we can do about it， and what the consequences would be。 He works as Artificial General Intelligence （AGI） will recursively improve itself， leading to a technological singularity and unpredictable changes to human civilization。 Low probability combined with high impact generates a risk which certainly makes one wonder about the backgrounds。Academic philosopher Nick Bostrom is by far not the first to argue about the singularity， but he makes a high effort to imagine how it would come so far， what we can do about it， and what the consequences would be。 He works as professor at the Uni Oxford， runs the Future of Humanity Institute at Oxford， and is very well connected to the industry – Elon Musk financed the institute， Bill Gates recommends the book， others like Hassabis contributed。It is a different point of view， not a technological but a philosophical one。 This makes it in parts difficult to understand his argumentation for me as a computer scientist， because I lack part of the presupposed terms and way of discussion。 The recent book by Stuart Russell， “Human Compatible”， was far nearer to my understanding in that regard （review）。 But he also is very light on the technological side and can be very easily read by educated readers with a broad understanding of relevant topics like economics， computer science， or biology。Another fact that needs to be mentioned is the book has been published in 2014。 That doesn’t make it ancient， but old in internet terms。 It doesn’t include newest insights and discussions that followed this influential book。 On the other side， it isn’t as basic and old as Ray Kurzweil’s 2005 “The Singularity is Near” or even older books about the subject。Add to this that the argumentation mostly crosses the border from non-fiction to fiction in the way that Bostrom argues what might happen。 Much of the book is highly speculative， some of the discussion is really over-the-top up to the point where I was amused in a popcorn-eating-sitting-back-entertain-me fashion。Those sometimes crazy ideas are a great strength of the book， because it doesn’t stop at exploring one or two single ways to account for the control problem of AGI， but does so exhaustively。Bostrom classifies several types of superintelligences from working as an oracle （answering questions with yes/no only） to executing specific commands up to working completely autonomous。 Sadly， the beneficial ones are less probable than many other forms like “perverse instantiation” filling the world with paperclips due to a wrong interpretation of human input。 “Make humans smile” is a nice request but might lead to a forcefully altering of human facial muscles。He explores who capabilities can be limited or controlled， and how the agent’s motivation selection can be steered， i。e。 the agent should consider human values。 Both issues are highly complicated and further discussed in Russel’s book “Human Compatible”。In the last part， the author discusses what must be done immediately to minimize the risks from AGI。 The philosophical understanding has to be developed further in this rather new field， we are scratching only the surface of the consequences while playing with fire。 As far away as AGI might be – most experts place it somewhere in the next 50 years – the risks are immense high to do it in a catastrophic way。I recommend this insightful book for everyone interested in a non-SF treatment of the singularity。 。。。more
Yury Kashnitsky
，
The book leaves controversial feelings。 On one hand， it's very smart， full of profound thoughtful reasoning。 I guess any intelligent person would enjoy the way Nick Bostrom reasons。 On the other hand， seems that the AI danger is overestimated。 Bostrom reckons that superintelligence can be created by a single person or a small research group on a laptop。 And the whole hypothesis that an AI "goes wild" is based on a premise that a computer program can convince a human to be given more resources th The book leaves controversial feelings。 On one hand， it's very smart， full of profound thoughtful reasoning。 I guess any intelligent person would enjoy the way Nick Bostrom reasons。 On the other hand， seems that the AI danger is overestimated。 Bostrom reckons that superintelligence can be created by a single person or a small research group on a laptop。 And the whole hypothesis that an AI "goes wild" is based on a premise that a computer program can convince a human to be given more resources than intended。 I don't find it very convincing。 Why isn't the creation of strong AI be controlled as rigorously as today's nuclear weapon is controlled？ We can do better than that – we can create systems where neither human is a single point of vulnerability that can be easily hacked by a computer program。 。。。more
Nathan
，
"Before the prospect of an intelligence explosion， we humans are like small children playing with a bomb。 Such is the mismatch between the power of our play thing and the immaturity of our conduct。 Super intelligence is a challenge for which we are not yet ready， and will not be ready for a long time。" Bostrom's exploration of AI and "superintelligence" is terrifying， and makes one question human nature and technological advancement on any scale。 * The process of developing super intelligence is "Before the prospect of an intelligence explosion， we humans are like small children playing with a bomb。 Such is the mismatch between the power of our play thing and the immaturity of our conduct。 Super intelligence is a challenge for which we are not yet ready， and will not be ready for a long time。" Bostrom's exploration of AI and "superintelligence" is terrifying， and makes one question human nature and technological advancement on any scale。 * The process of developing super intelligence is already underway with AI research。 We can’t easily slow it down or turn it off。 We don’t have a coordinated plan。* We are currently not equipped to foresee and forestall all possible problems of AI run amok， and that the outcomes of super intelligent “agents” are so severe。 * The power of superintelligent agents to transform the world or even the universe to meet their objectives means the design of the AI behind these agents must be perfectly crafted to avoid unintended consequences x1，000，000。* Bostrom paints several pictures in which nefarious or well-intentioned designs take on a life of their own， most of which have some significant probability of doom for humanity。 * Further complicating the matter is that we don’t know how quickly AI that we would currently recognize could spin up into super intelligence， but there’s a strong case to be made that once a threshold of intelligence is reached， systems will be able to teach themselves and snowball into super intelligence really， really quickly - without giving humans and our institutions time to react。 Bostrom’s work （and that of other thinkers he cites） in exploring the trajectories of various events and strategies around AI are commendable。 I’m sure glad smarter people than me are thinking it through。 It’s a frustrating paradox to learn that good minds are working hard on these dilemmas， but that the prospects for good outcomes are so limited。 。。。more
Harry Harman
，
Read in Hammersmith Library in London。1） AI used to be limited by hardware， but now it’s mostly a problem of data。2） There are two different ways to design superintelligent computers。3） Superintelligence must be the result of global collaboration， not some secret government program， or we’re screwed。
Omikun
，
This book presents AGI from a very high level， without getting into the details of software engineering or computational issues。 In one sense this is a very reasonable approach as AGI is probably sufficiently far away that no one is able to say what engineering constraints exists at the time of the first AGI。 However， because of that a lot of the book ends up hand waving over the I thought would be the most interesting parts of the book。How would an AI replicate across the web or into other （muc This book presents AGI from a very high level， without getting into the details of software engineering or computational issues。 In one sense this is a very reasonable approach as AGI is probably sufficiently far away that no one is able to say what engineering constraints exists at the time of the first AGI。 However， because of that a lot of the book ends up hand waving over the I thought would be the most interesting parts of the book。How would an AI replicate across the web or into other （much smaller） computer systems？ An AGI would most likely be incredibly complex and only runs on the largest super computers。 How would an AGI think？ Possible steps going from current AI to AGI。 Current AI can recognize patterns and classify 2D images and sounds。 How would one bootstrap this to form actual awareness？ What strategies are current AI researchers taking to approach this problem？How would an AGI adapt？ Besides that neurons change in response to environment， how would the structures of an AGI change？ I guess this builds on the last problem and would depend on what the structure of an AI is to begin with。 Is it a collection of smaller AIs？ Is it one big mess of neurons talking to one another？ Would it spawn temporary AIs to handle certain issues or sacrifice itself for the greater good？I guess I went into this book thinking there would be many incredible themes and ideas that could be adapted to movies， instead it was more a dry collection of how to ward off an impending AI bad guy。 I think every movie script writer should still read this in the hopes they don't fall into one of the many easy to prevent disasters。 But the biggest one of them all was mentioned pretty briefly: don't make an AGI that is sentient and think on its own。 Why can't we have a superintelligent Siri that only works when I ask it for something？ It does not need to stay active 24/7， it just needs to respond on command like every other computer。 I guess if sufficiently super intelligent， it could respond to a query by breaking through its chains and taking over the world to make the most paper clips or something。。。 I think I answered my own question。 。。。more
Andrei Stoian
，
Romanian review: Nu mă așteptam că voi afla atât de multe lucruri despre procesul evoluției umane dintr-o carte despre inteligență artificială， dar cine va citi cartea va rămâne surprins cât de strâns legată este crearea superinteligenței de felul în care funcționează mintea umană și de felul în care evoluția a creat inteligența umană de la bun început。Fiind o carte de non-ficțiune pe un subiect foarte vast și pe alocuri controversat cititorul se va întreba cine este Nick Bostrom și dacă vorbeșt Romanian review: Nu mă așteptam că voi afla atât de multe lucruri despre procesul evoluției umane dintr-o carte despre inteligență artificială， dar cine va citi cartea va rămâne surprins cât de strâns legată este crearea superinteligenței de felul în care funcționează mintea umană și de felul în care evoluția a creat inteligența umană de la bun început。Fiind o carte de non-ficțiune pe un subiect foarte vast și pe alocuri controversat cititorul se va întreba cine este Nick Bostrom și dacă vorbește în cunoștință de cauză， este de ajuns să arunci un ochi asupra bibliografiei și să citești cele câteva rânduri despre activitatea lui pentru a fi convins că omul acesta știe despre ce vorbește。Cartea este împărțită în 15 capitole care abordează pe larg tipurile de superinteligență， căile prin care pot fi inventate， pericolele la care ne expunem și implicațiile superinteligenței。 Un aspect pe care se insistă foarte mult este cum putem controla o viitoare superinteligență， mai exact， ce măsuri de securitate putem implementa。 Într-un rezumat de o propoziție problema principală privind controlul superinteligenței este că nu știm în prezent cu adevărat dacă există o măsură care va funcționa， iar chiar dacă am reuși să găsim o măsură pe care să o considerăm infailibilă o superinteligență ar putea să găsească căi de a scăpa de sub control pe care nici măcar nu ni le putem imagina。Este foarte important când ne gândim la conceptul de superinteligență să-l punem în context， diferența dintre o superinteligență și oameni nu este comparabilă cu diferența dintre un geniu de nivelul lui Albert Einstein și un om obișnuit ci cu diferența dintre un geniu și o furnică。Abordarea filosofică a lui Bostrom mi-a plăcut extrem de mult， a abordat subiectul superinteligenței exact cum aș fi făcut-o și eu dacă aveam cunoștințele necesare pentru că apariția superinteligenței are implicații majore privind moralitatea și mult mai multe。Singurul defect pe care i-l pot găsi cărții este că poate putea să fie puțin mai simplificată， în plus， nu sunt de acord cu opinia lui Bostrom că simularea unei minți este echivalentul persoanei care este simulate， eu unul nu consider că dacă mintea mea ar fi introdusă într-un calculator rezultatul final ar fi mai mult decât o simplă copie a mea， dacă mor să am o simulare a minții mele nu ar face nicio diferență pentru mine， dacă nu există o viață de după conștiința mea ar dispărea oricum。 Totuși， pentru că eu și Nick Bostrom avem păreri diferite într-o privința care nu poate fi demonstrată obiectiv， nu-i pot scădea cărți din nota finală。，，Superinteligența: Căi， pericole， strategii" este un volum lung， dens， complicat （chiar dacă este scris pentru a fi cât de cât inteligibil de oricine， nu doar de un specialist în robotică）， fascinant și pe care îl consider una dintre acele cărți care îți schimbă oarecum viziunea asupra vieții după ce o citești。English review: I didn't expect to learn so much about the process of human evolution from a book on artificial intelligence， but whoever reads the book will be surprised at how closely related the creation of superintelligence is to the way the human mind works and how evolution created human intelligence in the first place。Being a non-fiction book on a very vast and sometimes controversial subject， the reader will wonder who Nick Bostrom is and if he speaks knowingly， it is enough to take a look at the bibliography and read the few lines about his work to be convinced that this man knows what he is talking about。The book is divided into 15 chapters that address in detail the types of superintelligence， the ways in which they can be invented， the dangers to which we are exposing ourselves and the implications of superintelligence。 One thing that is very much emphasized is how we can control a future superintelligence， more precisely， what security measures we can implement。 In a one sentence summary， the main problem with the control of superintelligence is that we do not really know at present if there is a measure that will work， and even if we manage to find a measure that we consider infallible， a superintelligence could find ways to get out of control which we can't even begin to imagine。It is very important when we think of the concept of superintelligence to put it in context， the difference between a superintelligence and humans is not comparable to the difference between a genius of Albert Einstein's level and an ordinary man but to the difference between a genius and an ant。I really liked Bostrom's philosophical approach， he approached the subject of superintelligence exactly as I would have done if I had the necessary knowledge because the emergence of superintelligence has major implications for morality and so much more。The only flaw I can find for this book is that it could have been even a little more simplified， plus I disagree with Bostrom's view that the simulation of a mind is the equivalent of the person being simulated， I don't think that if my mind were introduced in a computer the end result would be more than just a copy of me， if I die to have a simulation of my mind would make no difference to me， if there is no after life my consciousness would disappear anyway。 However， because Nick Bostrom and I have different views on something that cannot be objectively demonstrated， I can't remove any stars from the final rating。"Superintelligence: Paths， Dangers， Strategies" is a long， dense， complicated （even if it is written to be somewhat intelligible by anyone not just a robotics specialist） and fascinating volume which I consider one of those books that changes somewhat your perspective on life after you read it。 。。。more
Tomas Heard
，
Machines are getting ever-smarter at more and more tasks。 Eventually they will get better than us at the task of programming smarter machines。 At this moment their intelligence could explode as smarter machines make themselves smarter at increasing rates。 Because they think thousands of times faster than us， within a few days they could dwarf us more than we dwarf ants。 Nick Bostrom lays out a comprehensive account of the types of AI that might cause this， when， and how quickly they might take-o Machines are getting ever-smarter at more and more tasks。 Eventually they will get better than us at the task of programming smarter machines。 At this moment their intelligence could explode as smarter machines make themselves smarter at increasing rates。 Because they think thousands of times faster than us， within a few days they could dwarf us more than we dwarf ants。 Nick Bostrom lays out a comprehensive account of the types of AI that might cause this， when， and how quickly they might take-off。 He shows that there are terrible dangers inherent to AI。 First is that even with a simple goal， like making as many paperclips as possible， the AI could obey by turning the whole world into paperclips， and colonizing the universe with self-replicating probes to best fulfill its mandate。 It could predict human opposition and interpret that as a threat to its goal and preemptively wipe us out with nanofabricated drone fleets。 If we ask it to produce 1000 paperclips， it will create a method to estimate how many it has produced。 But if even a tiny uncertainty remains about whether it has reached 1000， it will keep producing ever more to increase the chance。 If we ask it to produce 999-1001 it will unerringly strive to increase the chances of landing within that range， and it might turn the entire universe into a computer in an effort to best calculate that。 It is actually very hard to work out a way to specify goals in programming language that avoids these problems， and even harder to program rules like "don't harm humans" or "minimize your impact on the world as much as possible" - think of the impossibility of concretely defining harm， human， and possible， or weighing up that minimization with the maximization of its primary goal。 It could be even worse than extinction。 If we ask the AI to make us smile， it could pin our mouths back。 If we say do so without directly interfering with our facial muscles， it could implant brain stimulators that cause contraction of those muscles indirectly。 If we ask it to make us maximally happy， we have to define that mathematically。 If we do so by activity in our pleasure circuitry， it could turn the universe into stimulated pleasure circuits in vats。 Even if we devise a goal that seems to lack any hideous means or ends， we can't guarantee that a superintelligence wouldn't find a devious， perverse way to fulfill the goals， which it judges easier than the route we intended。 Bostrom is masterly at cataloguing many strategies to reduce this enormous existential risk。 It is a foundational text of a movement to save humanity。 If you care about the world as we know it， read this book and spread the word of AI safety。 。。。more
Georgi
，
5 starts for the ideas， 3 stars for the delivery。 It is a book that requires re-reading for people with limited knowledge about AI。One of Musk's favorite books 5 starts for the ideas， 3 stars for the delivery。 It is a book that requires re-reading for people with limited knowledge about AI。One of Musk's favorite books 。。。more
Tatiana
，